<html>
<head><meta charset="utf-8"><title>Test meta-coverage #86366 · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html">Test meta-coverage #86366</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="242884986"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242884986" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242884986">(Jun 16 2021 at 14:04)</a>:</h4>
<p>Do we have some way that we are tracking tests that are not exercised by the CI?</p>



<a name="242885249"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242885249" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242885249">(Jun 16 2021 at 14:05)</a>:</h4>
<p>Not that I'm aware of. Some tests have a comment explaining why they are disabled but I don't think that's been rigorously enforced.</p>



<a name="242885270"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242885270" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242885270">(Jun 16 2021 at 14:05)</a>:</h4>
<p>Currently I’m grepping for <code>x.py .* test</code> in <code>src/ci</code> and skimming the results. (And also taking the opposite direction of looking for the directory of interest there)</p>



<a name="242885308"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242885308" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242885308">(Jun 16 2021 at 14:05)</a>:</h4>
<p>specifically, I’m curious as to whether <code>src/test/rustdoc-ui</code> is tested in CI, since I’m currently seeing it fail on my local build</p>



<a name="242885348"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242885348" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242885348">(Jun 16 2021 at 14:05)</a>:</h4>
<p>Oh, you're talking about suites of tests not individual ones.</p>



<a name="242885356"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242885356" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242885356">(Jun 16 2021 at 14:05)</a>:</h4>
<p>Really, I mean whole test suites that are not exercised by the CI. :)</p>



<a name="242885366"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242885366" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242885366">(Jun 16 2021 at 14:06)</a>:</h4>
<p>Yeah exactly</p>



<a name="242885431"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242885431" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242885431">(Jun 16 2021 at 14:06)</a>:</h4>
<p>though tracking individual tests would be good too</p>



<a name="242885445"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242885445" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242885445">(Jun 16 2021 at 14:06)</a>:</h4>
<p>but the suites are in some ways more troubling</p>



<a name="242885492"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242885492" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242885492">(Jun 16 2021 at 14:06)</a>:</h4>
<p>I'm not aware of any tracking for that either.</p>



<a name="242885503"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242885503" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242885503">(Jun 16 2021 at 14:06)</a>:</h4>
<p>I mean, I understand the idea of saying “we want to archive these tests because they are useful for developers of those tools"</p>



<a name="242885545"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242885545" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242885545">(Jun 16 2021 at 14:06)</a>:</h4>
<p>but at the same time, its pretty frustrating to not know which test suites I should be expected to exercise locally</p>



<a name="242885572"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242885572" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242885572">(Jun 16 2021 at 14:07)</a>:</h4>
<p>in order to emulate the expectations of the CI</p>



<a name="242885626"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242885626" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242885626">(Jun 16 2021 at 14:07)</a>:</h4>
<p>(I suppose the “right answer” here is to run the same thing the CI does locally, via docker or whatever. But I’ve managed to get this far without doing that…)</p>



<a name="242885752"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242885752" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242885752">(Jun 16 2021 at 14:08)</a>:</h4>
<p>I don't personally have a good workflow for running our CI docker images so I've only reached for that when all else fails.</p>



<a name="242885820"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242885820" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242885820">(Jun 16 2021 at 14:08)</a>:</h4>
<p>and now that I'm mostly using Windows ... Docker is kind of a mess there</p>



<a name="242885888"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242885888" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242885888">(Jun 16 2021 at 14:09)</a>:</h4>
<p>okay well at least now I don’t feel like I’m alone in my lack of a docker-oriented workflow</p>



<a name="242886053"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242886053" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242886053">(Jun 16 2021 at 14:10)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> Chalk up another non-docker person here. <span aria-label="wave" class="emoji emoji-1f44b" role="img" title="wave">:wave:</span></p>



<a name="242887734"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242887734" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242887734">(Jun 16 2021 at 14:21)</a>:</h4>
<p>(The <a href="https://rustc-dev-guide.rust-lang.org/tests/intro.html#testing-with-docker-images">rustc dev guide</a> certainly makes it <em>sound</em> like docker is easy-peezy…)</p>



<a name="242889567"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242889567" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242889567">(Jun 16 2021 at 14:34)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> rustdoc-ui is definitely run on PR CI (even before bors CI). Is there a reason you think it doesn't?</p>



<a name="242889619"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242889619" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242889619">(Jun 16 2021 at 14:34)</a>:</h4>
<p>Because I’m seeing a local failure when I run it, against an unchanged build of the compiler</p>



<a name="242889679"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242889679" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242889679">(Jun 16 2021 at 14:35)</a>:</h4>
<p>What is the failure?</p>



<a name="242889684"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242889684" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242889684">(Jun 16 2021 at 14:35)</a>:</h4>
<p>and my attempts to grep for <code>rustdoc-ui</code> in <code>src/ci</code> met with no hits</p>



<a name="242889715"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242889715" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242889715">(Jun 16 2021 at 14:35)</a>:</h4>
<p>Right, it's run by default as part of x.py test</p>



<a name="242889733"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242889733" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242889733">(Jun 16 2021 at 14:35)</a>:</h4>
<p>where do we run <code>x.py test</code> with no arguments in the CI?</p>



<a name="242889826"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242889826" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242889826">(Jun 16 2021 at 14:36)</a>:</h4>
<p>I don't have time to look right now but search for <code>--exclude tidy</code></p>



<a name="242889893"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242889893" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242889893">(Jun 16 2021 at 14:36)</a>:</h4>
<p>Okay. I cannot recover the rustdoc-ui failure immediately because I started bisecting over our history to see where the failure was injected</p>



<a name="242889921"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242889921" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242889921">(Jun 16 2021 at 14:36)</a>:</h4>
<p>but I’ll report it somewhere when I see it again</p>



<a name="242894331"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242894331" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242894331">(Jun 16 2021 at 15:07)</a>:</h4>
<p>Many of the "test" builders just use the default <code>x.py test</code>.  <a href="https://github.com/rust-lang/rust/blob/8daad743c4bb941536860b4df20111cb71b8c777/src/ci/docker/host-x86_64/x86_64-gnu/Dockerfile#L28">Here</a> is an example of the basic linux builder.  <a href="https://github.com/rust-lang/rust/blob/8daad743c4bb941536860b4df20111cb71b8c777/.github/workflows/ci.yml#L308">Here</a> is macos.  <a href="https://github.com/rust-lang/rust/blob/8daad743c4bb941536860b4df20111cb71b8c777/.github/workflows/ci.yml#L330-L338">Windows</a> essentially does <code>x.py test</code>, but it is split across two jobs due to it being slow.</p>



<a name="242894726"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242894726" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242894726">(Jun 16 2021 at 15:10)</a>:</h4>
<p>Hm, now that Windows is nowhere near the slowest (looking at you dist-aarch64-apple), I wonder if it would make sense to merge the windows jobs back together.  They duplicate all the work of building the compiler and tools twice.</p>



<a name="242895675"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242895675" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242895675">(Jun 16 2021 at 15:16)</a>:</h4>
<p>Yes, I now understand/remember that my local pattern of running <code>x.py</code> with explicit arguments is to workaround <a href="https://github.com/rust-lang/rust/issues/47163">#47163</a>: "Debuginfo tests are not running"</p>



<a name="242905683"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242905683" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242905683">(Jun 16 2021 at 16:26)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116083">pnkfelix</span> <a href="#narrow/stream/131828-t-compiler/topic/Test.20meta-coverage/near/242889921">said</a>:</p>
<blockquote>
<p>but I’ll report it somewhere when I see it again</p>
</blockquote>
<p><span class="user-mention" data-user-id="232545">@Joshua Nelson</span> here is gist with failures:<br>
<a href="https://gist.github.com/pnkfelix/99e0203960c48994c2bf93064b0344c2">https://gist.github.com/pnkfelix/99e0203960c48994c2bf93064b0344c2</a></p>
<p>(I’m still bisecting but that was my latest “bad” run)</p>



<a name="242905753"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242905753" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242905753">(Jun 16 2021 at 16:27)</a>:</h4>
<p>Hmm I remember someone posting a similar issue on M1 recently</p>



<a name="242905770"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242905770" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242905770">(Jun 16 2021 at 16:27)</a>:</h4>
<p>It seems very strange that it only fails for some people</p>



<a name="242905871"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242905871" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242905871">(Jun 16 2021 at 16:28)</a>:</h4>
<p>I wonder if it's related to the default rustdoc toolchain somehow? Maybe the test suite is buggy and uses that instead of the in tree rustdoc?</p>



<a name="242906441"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242906441" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242906441">(Jun 16 2021 at 16:32)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> also I don't think bisecting will get you far, it will lead to the change that added the "compile fail" string</p>



<a name="242906477"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242906477" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242906477">(Jun 16 2021 at 16:33)</a>:</h4>
<p>Which isn't where the bug is, this must be a pre-existing bug in compile test for it to be inconsistent like this</p>



<a name="242906561"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242906561" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242906561">(Jun 16 2021 at 16:33)</a>:</h4>
<p>(alternatively, you can bisect only changes to rustdoc-ui and that should go a lot quicker)</p>



<a name="242906929"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242906929" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242906929">(Jun 16 2021 at 16:36)</a>:</h4>
<p><span class="user-mention silent" data-user-id="232545">Joshua Nelson</span> <a href="#narrow/stream/131828-t-compiler/topic/Test.20meta-coverage/near/242905871">said</a>:</p>
<blockquote>
<p>I wonder if it's related to the default rustdoc toolchain somehow? Maybe the test suite is buggy and uses that instead of the in tree rustdoc?</p>
</blockquote>
<p>This sounds like a promising line of inquiry</p>



<a name="242938060"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242938060" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242938060">(Jun 16 2021 at 20:26)</a>:</h4>
<p><del>hmm. Super-weird: As <span class="user-mention" data-user-id="232545">@Joshua Nelson</span> predicted, bisection ended up putting blame on PR <a href="https://github.com/rust-lang/rust/issues/84863">#84863</a>. But the <em>weird</em> part is that after bisection signalled that PR as “bad”, subsequent rebuilds atop that commit (after building with earlier commits as part of bisection process) have had no problems with compiletest output in the rustdoc-ui suite.</del> update: user fail. git bisect, upon finishing, put itself back onto the commit where I had initiated <code>git bisect</code></p>



<a name="242938145"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242938145" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242938145">(Jun 16 2021 at 20:27)</a>:</h4>
<p><del>I wonder if there’s something going on with cached state. I already did <code>rm -rf build/x86_64-unknown-linux-gnu/test/rustdoc-ui/</code> to try to account for any test artifacts being carried over from a “working” build, but I’m going to try again with a totally clean rebuild.</del></p>



<a name="242941736"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242941736" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242941736">(Jun 16 2021 at 20:53)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> try removing your default rustup toolchain so it's not possible for compiletest to use it by accident</p>



<a name="242941832"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242941832" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242941832">(Jun 16 2021 at 20:54)</a>:</h4>
<p>I don't think bisecting is going to help much</p>



<a name="242941898"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242941898" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242941898">(Jun 16 2021 at 20:54)</a>:</h4>
<p>I know, I just wanted to confirm</p>



<a name="242941924"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/242941924" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#242941924">(Jun 16 2021 at 20:54)</a>:</h4>
<p>and I didn’t know the PR offhand</p>



<a name="243020424"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/243020424" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#243020424">(Jun 17 2021 at 13:52)</a>:</h4>
<p><span class="user-mention" data-user-id="120518">@Eric Huss</span> ah, your insight here is key: The default for <code>x.py test</code> is implicitly <code>—stage 1</code>. But: the CI appears to explicitly pass <code>—stage 2</code> on all of its invocations of <code>x.py test</code></p>



<a name="243233030"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/243233030" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Hameer Abbasi <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#243233030">(Jun 19 2021 at 06:41)</a>:</h4>
<p>Is that equivalent to saying "we expect <code>rustc</code> to build on beta, but the testsuite only to pass on it built with itself on the current commit"?</p>



<a name="243244865"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/243244865" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#243244865">(Jun 19 2021 at 11:24)</a>:</h4>
<p>Yes.</p>



<a name="243244878"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/243244878" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#243244878">(Jun 19 2021 at 11:25)</a>:</h4>
<p>Well, that's what's tested currently. Clearly people expect it to pass with --stage 1.</p>



<a name="243393557"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/243393557" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#243393557">(Jun 21 2021 at 14:49)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116083">pnkfelix</span> <a href="#narrow/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366/near/243020424">said</a>:</p>
<blockquote>
<p><span class="user-mention silent" data-user-id="120518">Eric Huss</span> ah, your insight here is key: The default for <code>x.py test</code> is implicitly <code>—stage 1</code>. But: the CI appears to explicitly pass <code>—stage 2</code> on all of its invocations of <code>x.py test</code></p>
</blockquote>
<p>I’ve filed an MCP, <a href="https://github.com/rust-lang/compiler-team/issues/439">compiler-team#439</a>, to try to address this problem.</p>



<a name="243393652"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Test%20meta-coverage%20%2386366/near/243393652" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Test.20meta-coverage.20.2386366.html#243393652">(Jun 21 2021 at 14:50)</a>:</h4>
<p>(with associated zulip stream in <a class="stream-topic" data-stream-id="233931" href="/#narrow/stream/233931-t-compiler.2Fmajor-changes/topic/CI.20should.20exercise.20.28subset.20of.29.20tests.20unde.E2.80.A6.20compiler-team.23439">#t-compiler/major changes &gt; CI should exercise (subset of) tests unde… compiler-team#439</a> )</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>